On Friday Sept 14th 2007, a scheduled cruise on the Zephyr with the CTD vehicle was aborted. This note walks down the highlights of the event and the resultant Lessons Learned generated on Monday 17th at the regular status meeting.

 Event: 

On Friday 14th,  while at the dock on the Zephyr setting up for the cruise we encountered network problems in connecting to the AUV which was connected by cable to local network. We spent some time trying to diagnose the problem thinking that it had to do with an incorrect setting of our laptop ('dovekie') with respect to the ships network. This turned out to be a wiring problem with a recently installed router but one which nevertheless set us back by a good half hour. In the anxiety to get going, and while installing new binaries from 'dovekie' to trex2 (the onboard autonomy Linux box) on the AUV, the /bin directory of the Linux filesystem got deleted. This was doable since we had been till that time, been logging in as 'root' with unlimited privileges. The immediate thought was to replace the disk drive of trex2 on the vehicle by its duplicate on shore 'trex1' which is used for testing. However it turned out that the identical event had occurred the evening before and trex1 was also non-functional.

The immediate impact of this event was that we had to move the vehicle to the DMO AUV lab, remove the bubble and remove the hard drive of trex2. After substantial effort with a lot of help from Brent Roman and Kent Hedley (and Frederic on trex2), the /bin director was recovered and the two EPIC boards were brought on-line. However the delays this caused required a scrub of the cruise. On Monday 17th, we gathered for a Lessons Learned to see what general steps needed to be taken to ensure such serious errors are not repeated. The results are as below:

1, Disallow use of 'root' for deployment/development: The key and obvious finding was the process we had been using  to deploy (and developed) was unnecessary reliant on using the 'root' account on the two EPIC board computers. There is neither a need not a requirement for such use. Therefore this modus-operandi has been discontinued. Action: Use the general user 'dorado1' account similar to what the DMO AUV team uses.

2. Use of 'bin' as a T-REX directory name caused an unncessary mapping to a system level directory: Again there is no specific need to have directory names which overlap. Action: Rename /bin to /exec

3. Need to have the autonomy deployment laptop 'dovekie' be configured to ship and freewave ops: A major stress point was the need to ensure that 'dovekie' is able to talk to the AUV while on board the ship as well as when the vehicle is freely floating in the water. Dovekie is a Linux based computer identical to the autonomy EPIC boards een if substantially faster; further they have an identical copy of the sources should those need to be modified at sea. The network configurations have worked once in the past; however there was general lack of confidence that our original configurations were valid (which was incorrect) and coupled with the network outage mentioned in the preamble, seemed to reinforce that this issue was bigger than it really was. Action: Have automated Network Configuration which can deal with configuring the laptop from

• shore  to ship
• ship to Ops
• Ops to shore

4. The importance of regression tests is not be under-estimated: Just prior to the cruise it was known that all the standard regression tests were either not run or when run, some had failed in ways that were not clearly understood. For instance, a long mission was showing a segmentation fault well into 1000secs into its run on the EPIC boards. Action: Ensure there are nightly builds on a fast Linux box (e.g Threadfish) which will run

• EUROPA regression tests
• T-REX regression tests

5. The importance of running on the EPIC testbed is not be under-estimated: trex1 tests coupled with 'foobar' the QNX box with the AUV sim, can and does very reasonably simulate the conditions of T-REX when in the water. Running on trex1 is therefore very valuable especially in terms of understanding timing and performance of the system overall. Action: Therefore it is critical that prior to a cruise (and well before it) we

• run tests from the head of SVN checking out T-REX sources

• run nightly build on trex1 following the above. Even if slow it is a very good harbinger of any issues to come at sea.

6. Prior to deployment, and given the sparsity of our sea going days, dry runs are important: The deployment process should be fully walked thru in detail to make everyone feel comfortable on the day of the deployment and to ensure there are no last minute issues. The deployment itself requires some some minor additional steps. Action:

• Do a dry run/walk thru of the entire deployment process the day before and sans vehicle network test the full process to deploy. If the Zephyr is available at the dock the previous day, then Dovekie should be taken on board to test the network.
• Have a simple and small cheat-sheet to mark off what needs to be done; something akin to a check-list
• Have a large capacity USB Flash drive handy
• If possible install binaries on the AUV and practice deployment n the vehicle the previous day. Given the demand on the vehicle and the Zephyr this might not be as easy to do; but an attempt should be made if not the day immediately before, then sufficiently prior to the cruise day.

7. Need for a run-out sequence on the AMC: When T-REX fails or more likely spending substantial time churning while replanning especially with multiple reactors, we should consider executing a run-out sequence. Such a sequence for instance could keep the control within T-REX while turning the prop slowly or perhaps circling at an assigned depth similar to the standby mode of spacecraft. Such an eventuality should be executed only in the worst case of plan failure. Action: To do this, we could

• extend NAV failure timeout
• have a canned "smart" run out sequence on AMC side or
• return a version of the originally developed 'Idle' behavior

8. Bound the box of the navigational envelope of the vehicle: To ensure that the vehicle terminates should it inadvertently go outside a test area, a check in the model can ensure that when the vehicle comes up to the surface, it is guaranteed to be within the boundaries specified. Action: Requires model change and is associated with a run-out sequence mentioned above.